Skip to content

Improve missing-metric incident-onset evidence - #104

Open
JeffPengCoder wants to merge 3 commits into
NVIDIA:mainfrom
JeffPengCoder:fix/observability-incident-onset
Open

JeffPengCoder wants to merge 3 commits into
NVIDIA:mainfrom
JeffPengCoder:fix/observability-incident-onset

Conversation

@JeffPengCoder

@JeffPengCoder JeffPengCoder commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

Refs #103

Draft: make missing-metric onset investigation use source-sample evidence, not
the boundaries of the last chart query. This is not a claim that every model
or the complete native OpenClaw workflow now respects the skill.

An empty recent range does not establish when data stopped. There are also two
lower-level pitfalls: the current stride compaction can omit the final returned
point, and query_range evaluation times can continue past the last original
sample because of Prometheus lookback. Simply widening a chart query does not
resolve those ambiguities.

This change keeps one workflow together: historical evidence for a missing
metric, its read-only investigation tool, the skill instructions consuming that
evidence, and the runtime wiring and tests required to ship it.

  • Preserve both endpoints when compacting range results; disclose point counts,
    downsampling and returned boundary timestamps.
  • Mark generic range output as query evidence, not a determined incident onset.
    Add explicit ISO formatting for timestamp-valued queries without guessing that
    ordinary metric values are timestamps.
  • Add observability_metric_last_seen. It searches adjacent raw range vectors
    through the instant query API, retaining the metric and exact source labels.
    It returns the last original sample, actual checked windows, elapsed time, and
    an optional cadence-qualified first-missing-expected-sample estimate.
  • Bound work to 32 requests, 20 seconds overall, five seconds per request and
    2 MiB per response. Default history is seven days; maximum requested history
    is 30 days. Ambiguous series, errors, partial responses and unsupported result
    types do not become evidence of missing data.
  • Update the observability skill to actually backtrack on onset follow-ups,
    distinguish sample age from searched coverage, and keep entirely empty
    history unknown. No prior existence or retention claim is inferred.
  • Ship the module through the seed ConfigMap/init copy, declare the plugin tool
    contract, honor tool enable/disable controls, and classify it as read-only for
    View/automation policy. No new infrastructure mutation permission is added.

Validation

Local runtime: Node.js 25.8.1, Helm 3.19.0, Prometheus 3.16.0. Chart dependencies
use the public OpenShell 0.0.75 chart and the same minimal IRAOP stub as CI.

MOSAIC_HISTORY_TEST_URL=http://127.0.0.1:19094 \
  node --test utils/*.test.mjs helm/mosaic-stack/tests/*.test.mjs \
    tests/docs/*.test.mjs helm/afoa-inference/tests/*.test.mjs
# 280 tests: 278 passed, 0 failed, 2 skipped

helm lint helm/mosaic-stack
helm lint helm/afoa-inference
python3 docs/package_agent_docs.py --check
git diff --check

The two skips are native OpenClaw memory integration and a real VictoriaMetrics
endpoint. The real Prometheus history test was enabled and covered all four
synthetic cases. The fixture generator emits all 30121 original samples, with
no scrape targets or stale markers. Setup commands and every fixture row are in
helm/mosaic-stack/tests/metric-history.md.

Deterministic tests also cover adjacent (start,end] windows, a sample on a
window boundary, zero and non-finite values, unknown cadence, empty history,
request/time/size bounds, multiple series, label escaping, invalid input,
transport failures, partial warnings, malformed samples, module rendering and
read-only access controls.

Repeatable model/harness backtest

Add a portable runner and reproduction guide for comparing two source revisions with the same served model. It covers all four fixture cases plus the empty-window onset follow-up, with repeated runs and full request, response, tool, backend and answer records.

The runner validates all 30,121 fixture samples, records source/model/server identities and inference settings, and rejects unmatched comparisons. Normal inference-cache behavior is the default; cache isolation is a separately labeled diagnostic. Completion and tool invocation are execution observations, not factual-accuracy grades; every answer remains subject to the documented case-specific review.

Seven new runner tests pass. Both the PR base and candidate were additionally exercised against real Prometheus with an explicitly synthetic model stub to verify the replay plumbing. That smoke test does not measure a real model. The requested real Nemotron comparison has now completed, as recorded below. The older Qwen results remain separate evidence; native OpenClaw acceptance is still outstanding.

Nemotron 3 Super NVFP4: observed backtest

The complete run record and all final answers, token usage, verdicts and provenance are checked in. No case rows or failed attempts are silently removed.

  • Model: nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4, revision ff433f5493e25d631c9f12b5d55c674229923d02; this is 3 Super, not 3.5 Super VL.
  • Serving: DGX Spark / GB10, ARM64 vLLM 0.27.1; unchanged official ModelOpt NVFP4/FP8 layer policy, FP8 KV cache, qwen3_coder and checkpoint-pinned super_v3 parsers.
  • Settings: temperature 1, top_p 0.95, thinking disabled, seeds 1–3, 24 turns, 8,192 output tokens/turn, 180 seconds/request. Normal request-cache behavior; this server reports prefix caching disabled.
  • Source: baseline 307afa7e485a334389b4f2c8c68128fbf84ff007, candidate 8305d59e44470e10d268a4e60fc36870929b2e27. The subsequent result-record commit changes documentation only.
  • Coverage: all five cases × three seeds × both revisions, 30 canonical outcomes. One baseline transport failure was retained and replaced by one same-seed attempt; no answer-quality failures were rerun. Both smoke outcomes are recorded separately.
Observation Baseline Candidate
Completed answers with model-selected investigation 15/15 15/15
Whole-answer factual review passed 0/15 3/15
Correct original last-sample time in the nine gap answers 0/9 7/9
Correct zero-versus-absence distinction 2/3 3/3
Total model-selected tool calls 81 45
Tool errors 4 0

Codex reviewed the full answers against the pre-run rubric; this is not a blinded human evaluation or a general model accuracy score. The complete per-case table, exact durations, all per-turn token records and both score definitions are in the linked record. The suite uses a binary rubric, so its raw 0/1 mean equals its binary full-success rate.

The candidate improves source-sample retrieval, but it still sometimes stops a follow-up at an empty 48-hour window, omits the supplied cadence, conflates a last-present sample with the first expected absence, or adds unsupported workload/collector claims. One response recovered the right dates but contradicted itself with “three hours” versus “three days.” The three-day-gap seed-2 pass carries an explicit precision note in the record. These results support keeping this PR Draft and #103 open until native-runtime investigation completion and evidence-checked final output are verified. Thinking-enabled behavior was not measured.

Behavior evidence and draft limitations

An isolated model/tool replay used nvidia/Qwen3.6-35B-A3B-NVFP4, a fixed
investigation time of 2026-11-06T00:00:00Z, and the real synthetic Prometheus
fixture. The skill was explicitly loaded; this does not test native skill
selection, complete OpenClaw prompting, or the Mosaic UI.

With the same request-isolated inference control on both versions, the upstream
follow-up case incorrectly dated a three-day gap to the start of the recent
two-hour query, 2026-11-05T22:00:00Z. The candidate's two follow-up repetitions
both invoked the new tool, returned the actual last sample
2026-11-02T23:59:00Z, and qualified 2026-11-03T00:00:00Z as the first expected
missing sample at the known 60-second cadence, not a physical fault timestamp.

All seven final candidate replays completed after one investigation tool call
and two model turns, but not every prose claim passed review:

Case Repetition Observed result Remaining limitation
Three-day gap, direct 2 Correct original last sample and elapsed sample age Said cadence was unknown despite the environment supplying it; misstated the sum of four empty windows as 75 hours instead of 55
Three-day gap, follow-up after empty two-hour query 2 Correct last sample, cadence-qualified missing time and 79-hour checked coverage Native OpenClaw/UI behavior not tested
No available history 2 Correctly reported unknown onset over the checked 168 hours, without asserting an earlier outage Possible causes remain hypotheses, not established facts
Fresh zero 2 Queried real samples and correctly distinguished zero from absence Overstated an idle workload / healthy collector conclusion from utilization data
Two-hour gap 2 Correct original last sample and elapsed sample age Omitted known cadence in the call and then described it as unknown
Three-day gap, follow-up after empty two-hour query 3 Correct last sample and cadence-qualified missing time Native OpenClaw/UI behavior not tested
Three-day gap, direct 3 Correct original last sample, elapsed sample age and approximate onset Did not pass the known cadence into the tool; explained the resulting null estimate in prose

Earlier shared-cache replays also showed unstable tool use. An identical-input
six-request probe produced tool calls with fresh cache salts and on a warm
same-salt isolated prefix, while the two shared-prefix probes did not call a
tool. That is an inference-environment confound, not proof of the root cause of
#103. Both sides of the controlled comparison used a fresh per-request
cache_salt. No inference-server configuration or cache workaround is changed
in this PR, and raw model logs and deployment configuration are not included.

The new tool requires raw Prometheus range-vector semantics. It fails closed
for evaluated-vector substitutes and native histograms; it does not locate
threshold breaches or previous recovered gaps. Generic query tools remain
available for those investigations. Maintainer feedback on the dedicated
read-only tool/API is requested before treating this draft as merge-ready.

What remains before this can close the issue

This patch repairs the evidence-producing layer and adds a bounded investigation
path. It does not enforce the agent's decision to use that path or validate
every claim in its final natural-language answer. Passing tool tests and CI is
therefore not the same as fully addressing the reported behavior. Further work is feasible,
but should be verified in the actual conversation runtime:

  1. Native runtime investigation completion: exercise automatic skill
    selection and the complete Mosaic/OpenClaw conversation, particularly a
    follow-up after an empty recent query. The response must either be grounded
    in same-source historical evidence or identify an actual search/access limit.
    Merely suggesting a still-available historical query is not completion.
  2. Evidence-checked factual output: render or validate sample timestamps,
    elapsed age, checked coverage and any cadence-qualified missing-time estimate
    from structured tool evidence. Do not let the model substitute a query
    boundary, recompute a contradictory duration, claim retention from an empty
    result, or turn metric presence into proof of collector/workload health.
    This runtime/output enforcement is not implemented by the current patch.
  3. Repeated end-to-end regression: cover a multi-day gap, the empty-window
    follow-up, no available history, a fresh zero, a shorter gap, ambiguous
    sources and backend errors. Include the normal inference configuration, not
    only request-cache-isolated replay, and record remaining failures.

The observed inference-cache sensitivity should be investigated separately. It
is not evidence that the issue reporter had the same serving problem, and no
serving configuration change should be implied by merging this patch.

For now this PR references the issue without a closing keyword. Keep #103 open
until the native-runtime and factual-output acceptance criteria are satisfied;
the reference can then be converted to a closing link if maintainers agree.

Checklist

@JeffPengCoder
JeffPengCoder marked this pull request as ready for review November 9, 2026 21:59
Refs NVIDIA#103. Preserve range boundaries and source timestamps, add a bounded read-only last-seen investigation, and wire its skill, permissions, Helm assets, and synthetic regression coverage.

Signed-off-by: JeffPengCoder <jepeng@nvidia.com>
Signed-off-by: JeffPengCoder <jepeng@nvidia.com>
Signed-off-by: JeffPengCoder <jepeng@nvidia.com>
@JeffPengCoder
JeffPengCoder force-pushed the fix/observability-incident-onset branch from 415edc2 to 4e6c153 Compare November 19, 2026 06:10

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant